Dean Webb - Vehicle Detection & Tracking Pipeline¶
In this project, your goal is to write a software pipeline to detect vehicles in a video (start with the test_video.mp4 and later implement on full project_video.mp4), but the main output or product we want you to create is a detailed writeup of the project. Check out the writeup template for this project and use it as a starting point for creating your own writeup.
Project Goals:¶
The goals / steps of this project are the following:
- Perform a Histogram of Oriented Gradients (HOG) feature extraction on a labeled training set of images and train a classifier Linear SVM classifier
- Optionally, you can also apply a color transform and append binned color features, as well as histograms of color, to your HOG feature vector.
- Note: for those first two steps don't forget to normalize your features and randomize a selection for training and testing.
- Implement a sliding-window technique and use your trained classifier to search for vehicles in images.
- Run your pipeline on a video stream (start with the test_video.mp4 and later implement on full project_video.mp4) and create a heat map of recurring detections frame by frame to reject outliers and follow detected vehicles.
- Estimate a bounding box for vehicles detected.
Here are links to the labeled data for vehicle and non-vehicle examples to train your classifier. These example images come from a combination of the GTI vehicle image database, the KITTI vision benchmark suite, and examples extracted from the project video itself. You are welcome and encouraged to take advantage of the recently released Udacity labeled dataset to augment your training data.
Some example images for testing your pipeline on single frames are located in the test_images folder. To help the reviewer examine your work, please save examples of the output from each stage of your pipeline in the folder called ouput_images, and include them in your writeup for the project by describing what each image shows. The video called project_video.mp4 is the video your pipeline should work well on.
Dataset - Load Data and Extract Features¶
In [1]:
%matplotlib inline
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import cv2
import glob
from PIL import Image
import time
import os
import pickle
from skimage.feature import hog
from sklearn.svm import LinearSVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
In [2]:
## Dataset Parameters ##
LABELS_CSV = 'FILL_LATER.csv'
TRAINING_DATASET_DIRECTORY = 'training_set/'
PIPELINE_SETUP_DIRECTORY = 'pipeline_setup_images/'
WORKING_DIRECTORY = 'data/'
NON_VEHICLES_TOKEN = 'non-vehicles'
dataset_path = "{}{}{}{}".format(WORKING_DIRECTORY, TRAINING_DATASET_DIRECTORY,'**/', '*.png')
DATACACHE_DIRECTORY = os.path.join(WORKING_DIRECTORY, 'datacache/')
## Image Processing ##
DEFAULT_LENGTH, DEFAULT_WIDTH, DEFAULT_DEPTH = (64, 64, 3)
if DEFAULT_DEPTH > 1:
DEFAULT_RESOLUTION = (DEFAULT_LENGTH, DEFAULT_WIDTH, DEFAULT_DEPTH)
else:
DEFAULT_RESOLUTION = (DEFAULT_LENGTH, DEFAULT_WIDTH)
## Feature Extraction Parameters ##
# Spatial Binning
SPATIAL = 32
BIN_SPATIAL_SIZE = (SPATIAL, SPATIAL)
# Color Histogram
HIST_NBINS = 128
COLOR_SPACE = 'HSV'
# HOG Parameters
HOG_ORIENTATIONS = 9
HOG_PIXELS_PER_CELL = 8
HOG_CELLS_PER_BLOCK = 2
HOG_CHANNEL = 'ALL' # Can be 0, 1, 2, or "ALL"
# SVC Parameters
VALIDATION_PORTION = .3
N_PREDICTIONS = 100
In [3]:
# Define a function to scale .PNG and JPEG Files both to 0 to 1
def normalize_pixels(img):
max_pixel_value = np.max(img)
if max_pixel_value > 1.0:
img = np.copy(np.multiply(img, 1.0 / 255.0)).astype(np.float64)
return img
# Define a function to scale .PNG and JPEG Files both to 0 to 1
def denormalize_pixels(img):
max_pixel_value = np.max(img)
if max_pixel_value <= 1.0:
img = np.copy(np.multiply(img, 255.0)).astype(np.float64)
return img
In [4]:
# Define a function to compute binned color features
def bin_spatial(img, size=BIN_SPATIAL_SIZE):
color1 = cv2.resize(img[:,:,0], size).ravel()
color2 = cv2.resize(img[:,:,1], size).ravel()
color3 = cv2.resize(img[:,:,2], size).ravel()
return np.hstack((color1, color2, color3))
In [5]:
# Define a function to compute color histogram features
def color_hist(img, nbins=HIST_NBINS):
# Compute the histogram of the color channels separately
channel1_hist = np.histogram(img[:,:,0], bins=nbins)
channel2_hist = np.histogram(img[:,:,1], bins=nbins)
channel3_hist = np.histogram(img[:,:,2], bins=nbins)
# Concatenate the histograms into a single feature vector
hist_features = np.concatenate((channel1_hist[0], channel2_hist[0], channel3_hist[0]))
return hist_features
In [6]:
# Define a function to return HOG features and visualization --
def get_hog_features(img, orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL,
cell_per_block=HOG_CELLS_PER_BLOCK, vis=False, feature_vec=True):
denormalized_img = denormalize_pixels(img)
if vis == True:
features, hog_image = hog(denormalized_img, orientations=orient, pixels_per_cell=(pix_per_cell, pix_per_cell),
cells_per_block=(cell_per_block, cell_per_block), transform_sqrt=False,
visualise=vis, feature_vector=feature_vec)
return features, hog_image
else:
features = hog(denormalized_img, orientations=orient, pixels_per_cell=(pix_per_cell, pix_per_cell),
cells_per_block=(cell_per_block, cell_per_block), transform_sqrt=False,
visualise=vis, feature_vector=feature_vec)
return features
In [7]:
# Define a function to extract features from a list of images
def extract_features(imgs, cspace=COLOR_SPACE, spatial_size=BIN_SPATIAL_SIZE,
hist_bins=HIST_NBINS):
# Create a list to append feature vectors
features = []
for file in imgs:
image = mpimg.imread(file)
# Image read in from mpimg + .png -> (0 to 1) scaled
if cspace != 'RGB':
if cspace == 'HSV':
feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
elif cspace == 'LUV':
feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2LUV)
elif cspace == 'HLS':
feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2HLS)
elif cspace == 'YUV':
feature_image = cv2.cvtColor(image, cv2.COLOR_RGB2YUV)
else: feature_image = np.copy(image)
# Apply bin_spatial() to get spatial color features
spatial_features = bin_spatial(feature_image, size=spatial_size)
# Apply color_hist() also with a color space option now
hist_features = color_hist(feature_image, nbins=hist_bins)
# Call get_hog_features() with vis=False, feature_vec=True
hog_image = np.copy(image)
hog_shape = np.asarray(hog_image.shape)
if HOG_CHANNEL == 'ALL':
hog_features = []
for channel in range(len(hog_shape)):
hog_features.append(get_hog_features(hog_image[:,:,channel]))
hog_features = np.ravel(hog_features)
else:
hog_features = get_hog_features(hog_image[:,:,HOG_CHANNEL])
# Append the new feature vector to the features list
features.append(np.concatenate((spatial_features, hist_features, hog_features)))
# Return list of feature vectors
return features
In [8]:
## Starting Training Pipeline ##
# Load Image Paths
images = glob.glob(dataset_path, recursive=True)
cars = []
notcars = []
for image in images:
if NON_VEHICLES_TOKEN in image:
notcars.append(image)
else:
cars.append(image)
print('Number of Vehicle Images Found:',len(cars),'(images in directory):', dataset_path)
print('Number of Non-Vehicle Images Found:',len(notcars),'(images in directory):', dataset_path)
assert len(images) == len(cars) + len(notcars), 'The subarrays have not split the dataset correctly.'
In [9]:
# Start Pipeline - Combine and Normalilze Features
car_features = extract_features(cars)
notcar_features = extract_features(notcars)
# Create an array stack of feature vectors
X = np.vstack((car_features, notcar_features)).astype(np.float64)
# Fit a per-column scaler
X_scaler = StandardScaler().fit(X)
# Apply the scaler to X
scaled_X = X_scaler.transform(X)
car_ind = np.random.randint(0, len(cars))
# Plot an example of raw and scaled features
fig = plt.figure(figsize=(12,4))
plt.subplot(131)
plt.imshow(mpimg.imread(cars[car_ind]))
plt.title('Original Image')
plt.subplot(132)
plt.plot(X[car_ind])
plt.title('Raw Features')
plt.subplot(133)
plt.plot(scaled_X[car_ind])
plt.title('Normalized Features')
fig.tight_layout()
print('Feature Vector size for Cars:', len(car_features[car_ind]))
print('Using Spatial Binning of:',BIN_SPATIAL_SIZE[0],
'and', HIST_NBINS,'histogram bins')
In [10]:
# Define the labels vector
y = np.hstack((np.ones(len(car_features)), np.zeros(len(notcar_features))))
In [11]:
# Split up data into randomized training and test sets
rand_state = np.random.randint(0, 100)
X_train, X_test, y_train, y_test = train_test_split(
scaled_X, y, test_size=VALIDATION_PORTION, random_state=rand_state)
print('Feature vector length:', len(X_train[0]))
In [12]:
# Use a linear SVC
svc = LinearSVC()
# Check the training time for the SVC
t=time.time()
svc.fit(X_train, y_train)
t2 = time.time()
print(round(t2-t, 2), 'Seconds to train SVC...')
# Check the score of the SVC
print('Test Accuracy of SVC = ', round(svc.score(X_test, y_test), 4))
# Check the prediction time for a single sample
t=time.time()
n_predict = N_PREDICTIONS
print('SVC predicts: ', svc.predict(X_test[0:n_predict]))
print('For these',n_predict, 'labels: ', y_test[0:n_predict])
t2 = time.time()
print(round(t2-t, 5), 'Seconds to predict', n_predict,'labels with SVC')
In [13]:
#Save Support Vector Classifier to Datacache
def save_to_datacache(support_vector_classifier, datacache_dir=DATACACHE_DIRECTORY):
os.makedirs(datacache_dir, exist_ok=True)
svc_pickle = os.path.join(datacache_dir,"svc_pickle.p")
if not os.path.exists(svc_pickle):
svc_hyperparameters = {'svc': svc,
'X_scaler':X_scaler,
'SPATIAL': SPATIAL,
'HIST_NBINS': HIST_NBINS,
'COLOR_SPACE': COLOR_SPACE,
'HOG_ORIENTATIONS': HOG_ORIENTATIONS,
'HOG_PIXELS_PER_CELL': HOG_PIXELS_PER_CELL,
'HOG_CELLS_PER_BLOCK': HOG_CELLS_PER_BLOCK,
'HOG_CHANNEL': HOG_CHANNEL
}
pickle.dump(svc_hyperparameters, open(svc_pickle, "wb"))
# Save classifier and parameters to datacache directory
save_to_datacache(svc)
Finished Training SVC - Now to Utilize Sliding Windows¶
In [14]:
import matplotlib.image as mpimg
import numpy as np
import cv2
from skimage.feature import hog
In [15]:
# Sliding Window Constants
WORKING_DIRECTORY = 'data/'
DATACACHE_DIRECTORY = os.path.join(WORKING_DIRECTORY, 'datacache/')
svc_pickle = os.path.join(DATACACHE_DIRECTORY,"svc_pickle.p")
TESTING_DATASET_DIRECTORY = 'testing_dataset/'
TESTING_PIPELINE_SETUP_DIR= 'test_images/'
testset_path = "{}{}{}".format(WORKING_DIRECTORY, TESTING_PIPELINE_SETUP_DIR, '*.jpg')
with open(svc_pickle, mode='rb') as f:
svc_hyperparameters = pickle.load(f)
## Feature Extraction Parameters ##
# Spatial Binning
SVC = svc_hyperparameters['svc']
X_SCALER = svc_hyperparameters['X_scaler']
SPATIAL = svc_hyperparameters['SPATIAL']
BIN_SPATIAL_SIZE = (SPATIAL, SPATIAL)
# Color Histogram
HIST_NBINS = svc_hyperparameters['HIST_NBINS']
COLOR_SPACE = svc_hyperparameters['COLOR_SPACE']
# HOG Parameters
HOG_ORIENTATIONS = svc_hyperparameters['HOG_ORIENTATIONS']
HOG_PIXELS_PER_CELL = svc_hyperparameters['HOG_PIXELS_PER_CELL']
HOG_CELLS_PER_BLOCK = svc_hyperparameters['HOG_CELLS_PER_BLOCK']
HOG_CHANNEL = svc_hyperparameters['HOG_CHANNEL'] # Can be 0, 1, 2, or "ALL"
BBOX_COLOR = (0, 0, 255)
BBOX_THICK = 6
## Sliding Windows Parameters - Horizon ##
SW_XSTART_STOPS = [(None, None), (None, None), (200, 1000),(400, 1300)]
SW_YSTART_STOPS = [(420, 650), (400, 575), (375, 500), (400,556)]
SW_XY_WINDOWS = [(240, 150), (120,96), (60,48), (96,96)]
SW_XY_OVERLAPS = [(0.75, 0.55), (0.75,0.5), (0.75,0.5), (0.5,0.5)]
SW_SPATIAL_FEAT_FLAG = True
SW_HOG_FEAT_FLAG = True
SW_COLOR_HIST_FEAT_FLAG = True
In [16]:
# Define a function to draw bounding boxes
def draw_boxes(img, bboxes, color=BBOX_COLOR, thick=BBOX_THICK):
# Make a copy of the image
imcopy = np.copy(img)
# Iterate through the bounding boxes
for bbox in bboxes:
# Draw a rectangle given bbox coordinates
cv2.rectangle(imcopy, bbox[0], bbox[1], color, thick)
return imcopy
In [17]:
# Define a function that takes an image, a list of start_stop positions in both x and y,
# a list of window sizes (x and y dimensions), and a list of overlap fraction (for both x and y)
def slide_windows(img, x_start_stops=[[None, None]], y_start_stops=[[None, None]],
xy_windows=[(64, 64)], xy_overlaps=[(0.5, 0.5)]):
windows = []
for i in range(len(x_start_stops)):
if len(x_start_stops) == len(xy_windows) and len(x_start_stops) == len(xy_overlaps):
windows = slide_window(img, np.asarray(x_start_stops[i]), np.asarray(y_start_stops[i]),
np.asarray(xy_windows[i]), np.asarray(xy_overlaps[i]), windows)
else:
windows = slide_window(img, np.asarray(x_start_stops[i]), np.asarray(y_start_stops[i]),
np.asarray(xy_windows[0]), np.asarray(xy_overlaps[0]), windows)
return windows
In [18]:
# Define a function that takes an image, start and stop positions in both x and y,
# window size (x and y dimensions), and overlap fraction (for both x and y)
def slide_window(img, x_start_stop=[None, None], y_start_stop=[None, None],
xy_window=(64, 64), xy_overlap=(0.5, 0.5), window_list=[]):
# If x and/or y start/stop positions not defined, set to image size
if x_start_stop[0] == None:
x_start_stop[0] = 0
# elif x_start_stop[0] >= img.shape[1]:
# x_start_stop[0] = img.shape[1] - 64
if x_start_stop[1] == None or x_start_stop[1] >= img.shape[1]:
x_start_stop[1] = img.shape[1]
if y_start_stop[0] == None:
y_start_stop[0] = 0
# elif y_start_stop[0] >= img.shape[0]:
# y_start_stop[0] = img.shape[0] - 64
if y_start_stop[1] == None or y_start_stop[1] >= img.shape[0]:
y_start_stop[1] = img.shape[0]
# Compute the span of the region to be searched
xspan = x_start_stop[1] - x_start_stop[0]
yspan = y_start_stop[1] - y_start_stop[0]
# Compute the number of pixels per step in x/y
nx_pix_per_step = np.int(xy_window[0]*(1 - xy_overlap[0]))
ny_pix_per_step = np.int(xy_window[1]*(1 - xy_overlap[1]))
# Compute the number of windows in x/y
nx_buffer = np.int(xy_window[0]*(xy_overlap[0]))
ny_buffer = np.int(xy_window[1]*(xy_overlap[1]))
nx_windows = np.int((xspan-nx_buffer)/nx_pix_per_step)
ny_windows = np.int((yspan-ny_buffer)/ny_pix_per_step)
# Loop through finding x and y window positions
for ys in range(ny_windows):
for xs in range(nx_windows):
# Calculate window position
startx = xs*nx_pix_per_step + x_start_stop[0]
endx = startx + xy_window[0]
starty = ys*ny_pix_per_step + y_start_stop[0]
endy = starty + xy_window[1]
# Append window position to list
window_list.append(((startx, starty), (endx, endy)))
return window_list
In [19]:
#Define a function for plotting multiple images
def visualize(fig, rows, cols, imgs, titles):
for i, img in enumerate(imgs):
plt.subplot(rows, cols, i+1)
plt.title(i+1)
img_dims = len(img.shape)
if img_dims < 3:
plt.imshow(img, cmap='hot')
plt.title(titles[i])
else:
plt.imshow(img)
plt.title(titles[i])
In [20]:
# Define a function to extract features from a single image window
# This function is very similar to extract_features()
# just for a single image rather than list of images
def single_img_features(img, color_space=COLOR_SPACE, spatial_size=(32, 32),
hist_bins=32, orient=9,
pix_per_cell=8, cell_per_block=2, hog_channel=0,
spatial_feat=True, hist_feat=True, hog_feat=True):
#1) Define an empty list to receive features
img_features = []
#2) Apply color conversion if other than 'RGB'
if color_space != 'RGB':
if color_space == 'HSV':
feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
elif color_space == 'LUV':
feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2LUV)
elif color_space == 'HLS':
feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
elif color_space == 'YUV':
feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
elif color_space == 'YCrCb':
feature_image = cv2.cvtColor(img, cv2.COLOR_RGB2YCrCb)
else: feature_image = np.copy(img)
#3) Compute spatial features if flag is set
if spatial_feat == True:
spatial_features = bin_spatial(feature_image, size=spatial_size)
#4) Append features to list
img_features.append(spatial_features)
#5) Compute histogram features if flag is set
if hist_feat == True:
hist_features = color_hist(feature_image, nbins=hist_bins)
#6) Append features to list
img_features.append(hist_features)
#7) Compute HOG features if flag is set
if hog_feat == True:
if hog_channel == 'ALL':
hog_features = []
for channel in range(feature_image.shape[2]):
hog_features.extend(get_hog_features(feature_image[:,:,channel],
orient, pix_per_cell, cell_per_block,
vis=False, feature_vec=True))
else:
hog_features = get_hog_features(feature_image[:,:,hog_channel], orient,
pix_per_cell, cell_per_block, vis=False, feature_vec=True)
#8) Append features to list
img_features.append(hog_features)
#9) Return concatenated array of features
return np.concatenate(img_features)
In [21]:
# Define a function you will pass an image
# and the list of windows to be searched (output of slide_windows())
def search_windows(img, windows, svc=SVC, X_scaler=X_SCALER, color_space=COLOR_SPACE,
spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS,
orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG,
hist_feat=SW_COLOR_HIST_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG):
#1) Create an empty list to receive positive detection windows
on_windows = []
#2) Iterate over all windows in the list
for window in windows:
#3) Extract the test window from original image
if window[0][0] <= 0:
x_start = 0
else:
x_start = window[0][0]
if window[1][0] >= img.shape[1]:
x_stop = img.shape[1]
else:
x_stop = window[1][0]
if window[0][1] <= 0:
y_start = 0
else:
y_start = window[0][1]
if window[1][1] >= img.shape[0]:
y_stop = img.shape[0]
else:
y_stop = window[1][1]
#print('Window Dims:', window)
#print('window[0][0]:', window[0][0])
#print('window[1][0]:',window[1][0])
#print('img[window[0][1]:window[1][1], window[0][0]:window[1][0]] shape:', img[window[0][1]:window[1][1], window[0][0]:window[1][0]].shape)
test_img = cv2.resize(img[y_start:y_stop, x_start:x_stop], (DEFAULT_LENGTH, DEFAULT_WIDTH))
#4) Extract features for that window using single_img_features()
features = single_img_features(test_img, color_space=color_space,
spatial_size=spatial_size, hist_bins=hist_bins,
orient=orient, pix_per_cell=pix_per_cell,
cell_per_block=cell_per_block,
hog_channel=hog_channel, spatial_feat=spatial_feat,
hist_feat=hist_feat, hog_feat=hog_feat)
#5) Scale extracted features to be fed to classifier
test_features = X_scaler.transform(np.array(features).reshape(1, -1))
#6) Predict using your classifier
prediction = svc.predict(test_features)
#7) If positive (prediction == 1) then save the window
if prediction == 1: # Car detected
on_windows.append(window)
heatmap[window[0][1]:window[1][1], window[0][0]:window[1][0]] +=1
#8) Return windows for positive detections
return on_windows, heatmap
In [22]:
# Try Scaling Windows on Test Images
image_paths = glob.glob(testset_path, recursive=True)
print('Found',len(image_paths),'images in directory:', testset_path)
In [ ]:
In [23]:
from scipy.ndimage.measurements import label
def apply_threshold(heatmap, threshold):
# Zero out pixels below the threshold
heatmap[heatmap <= threshold] = 0
return heatmap
def draw_labeled_bboxes(img, labels):
# Iterate through all detected cars
for car_number in range(1, labels[1] + 1):
#Find pixels with each car_number label value
nonzero = (labels[0] == car_number).nonzero()
# Identify x and y values of those pixels
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
#Define a bounding box based on min/max x and y
bbox = ((np.min(nonzerox), np.min(nonzeroy)), (np.max(nonzerox), np.max(nonzeroy)))
#Draw the box on the image
cv2.rectangle(img, bbox[0], bbox[1], BBOX_COLOR, BBOX_THICK)
# Return the image
return img
In [24]:
# Define a single function that can extract features using hog sub-sampling and make predictions
out_images = []
out_titles = []
## Sliding Windows Parameters ##
SW_XSTART_STOPS = [(None, None), (None, None), (192, 1216),(None, None), (200,None)]#, (400, 1300),(200, 1000),(400, 1300)]
SW_YSTART_STOPS = [(192, 640), (416, 640),(384, 576),(416, 640), (384, 512)]#, (500, 600), (375, 500), (400,656)]
SW_XY_WINDOWS = [(256, 128), (96,96), (128, 96), (96, 64), (64,64)]#, (96,64), (64,48), (96,96)]
SW_XY_OVERLAPS = [(0.75, 0.25),(0.750,0.50), (0.750, 0.350), (0.750, 0.500), (.75,.75)]#, (0.75,0.5), (0.75,0.5), (0.5,0.5)]
for img_path in image_paths:
t1 = time.time()
img = mpimg.imread(img_path)
draw_img = np.copy(img)
img = normalize_pixels(img).astype(np.float32)
#Make a heatmap of zeros
heatmap = np.zeros_like(img[:,:,0])
windows = []
hot_windows = []
threshold = 7
filename = os.path.split(img_path)[-1]
windows_1 = slide_windows(img, x_start_stops=SW_XSTART_STOPS, y_start_stops=SW_YSTART_STOPS,
xy_windows=SW_XY_WINDOWS,
xy_overlaps=SW_XY_OVERLAPS)
hot_windows_1, heatmap = search_windows(img, windows_1, SVC, X_SCALER, color_space=COLOR_SPACE,
spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS, orient=HOG_ORIENTATIONS,
pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG,
hist_feat=SW_COLOR_HIST_FEAT_FLAG)
print('BBoxes Found:', len(hot_windows_1))
windows.extend(windows_1)
hot_windows.extend(hot_windows_1)
window_img = draw_boxes(draw_img, hot_windows, color=BBOX_COLOR, thick=BBOX_THICK)
labels = label(apply_threshold(heatmap, threshold))
# Draw bounding boxes on a copy of the image
window_img_thresh = draw_labeled_bboxes(draw_img, labels)
out_images.append(window_img_thresh)
out_titles.append(filename)
out_images.append(apply_threshold(heatmap, threshold))
out_titles.append(filename)
print(time.time()-t1, 'seconds to process one image search', len(windows), 'windows')
fig = plt.figure(figsize=(12,24))
visualize(fig, 8, 2, out_images, out_titles)
In [ ]:
In [ ]:
Explore HOG Sub-Sampling Window Search¶
We want a more efficient way to detect vehicles. This approach will allow for only a single call to get HOG features. The pipeline will then find a sub sample
In [25]:
SW_CONVERT_COLOR = 'RGB2YCrCb'
def convert_color(img, conv=SW_CONVERT_COLOR):
if conv == 'RGB2YCrCb':
return cv2.cvtColor(img, cv2.COLOR_RGB2YCrCb)
if conv == 'BGR2YCrCb':
return cv2.cvtColor(img, cv2.COLOR_BGR2YCrCb)
if conv == 'RGB2LUV':
return cv2.cvtColor(img, cv2.COLOR_RGB2LUV)
In [26]:
# Define a single function that can extract features using hog sub-sampling and make predictions
out_images = []
out_maps = []
out_titles = []
out_boxes = []
## Sliding Window Parameters - HOG Sub-Sampling ##
SW_YSTART = 400
SW_YSTOP = 665
SW_SCALE = 1.75
ystart = SW_YSTART
ystop = SW_YSTOP
scale = SW_SCALE
spatial_size=BIN_SPATIAL_SIZE
hist_bins=HIST_NBINS
orient=HOG_ORIENTATIONS
pix_per_cell=HOG_PIXELS_PER_CELL
cell_per_block=HOG_CELLS_PER_BLOCK
hog_channel=HOG_CHANNEL
spatial_feat=SW_SPATIAL_FEAT_FLAG
hog_feat=SW_HOG_FEAT_FLAG
hist_feat=SW_COLOR_HIST_FEAT_FLAG
#Iterate over the test images
for img_path in image_paths:
img_boxes = []
t1 = time.time()
count = 0
img = mpimg.imread(img_path)
draw_img = np.copy(img)
#Make a heatmap of zeros
heatmap = np.zeros_like(img[:,:,0])
img = normalize_pixels(img).astype(np.float32)
img_to_search = img[ystart:ystop,:,:]
ctrans_tosearch = convert_color(img_to_search, conv=SW_CONVERT_COLOR)
if scale != 1:
imshape = ctrans_tosearch.shape
ctrans_tosearch = cv2.resize(ctrans_tosearch, (np.int(imshape[1]/scale), np.int(imshape[0]/scale)))
ch1 = ctrans_tosearch[:,:,0]
ch2 = ctrans_tosearch[:,:,1]
ch3 = ctrans_tosearch[:,:,2]
# Define blocks and steps as above
nxblocks = (ch1.shape[1] // pix_per_cell)-1
nyblocks = (ch1.shape[0] // pix_per_cell)-1
nfeat_per_block = orient*cell_per_block**2
window = HOG_PIXELS_PER_CELL*HOG_PIXELS_PER_CELL # 8 cells and 8 pix per cell
nblocks_per_window = (window // pix_per_cell)-1 # The // division is used for integers (for indices)
cells_per_step = 2 # Instead of overlap, define how many cells to step
nxsteps = (nxblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
nysteps = (nyblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
# Compute individual channel HOG features for the entire image
hog1 = get_hog_features(ch1, orient, pix_per_cell, cell_per_block, feature_vec=False)
hog2 = get_hog_features(ch2, orient, pix_per_cell, cell_per_block, feature_vec=False)
hog3 = get_hog_features(ch3, orient, pix_per_cell, cell_per_block, feature_vec=False)
for xb in range(nxsteps):
for yb in range(nysteps):
count += 1
ypos = yb*cells_per_step
xpos = xb*cells_per_step
# Extract HOG for this particular patch
hog_feat1 = hog1[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_feat2 = hog2[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_feat3 = hog3[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_features = np.hstack((hog_feat1, hog_feat2, hog_feat3))
xleft = xpos*pix_per_cell
ytop = ypos*pix_per_cell
# Extract the image patch
subimg = cv2.resize(ctrans_tosearch[ytop:ytop+window, xleft:xleft+window], (DEFAULT_LENGTH, DEFAULT_WIDTH))
# Get color features
spatial_features = bin_spatial(subimg, size=BIN_SPATIAL_SIZE)
hist_features = color_hist(subimg, nbins=HIST_NBINS)
test_feats = np.hstack((spatial_features, hist_features, hog_features))
# Scale features and make a prediction
test_features = X_SCALER.transform(test_feats.reshape(1, -1))
test_prediction = SVC.predict(test_features)
if test_prediction == 1:
xbox_left = np.int(xleft*scale)
ytop_draw = np.int(ytop*scale)
win_draw = np.int(window*scale)
cv2.rectangle(draw_img,(xbox_left, ytop_draw+ystart),
(xbox_left+win_draw, ytop_draw+win_draw+ystart),BBOX_COLOR,6)
img_boxes.append(((xbox_left, ytop_draw+ystart),
(xbox_left+win_draw,ytop_draw+win_draw+ystart)))
heatmap[ytop_draw+ystart:ytop_draw+win_draw+ystart, xbox_left:xbox_left+win_draw] +=1
print(time.time()-t1, 'seconds to run, total windows = ', count)
out_images.append(draw_img)
out_titles.append(os.path.split(img_path)[-1])
out_titles.append(os.path.split(img_path)[-1])
out_images.append(heatmap)
out_maps.append(heatmap)
out_boxes.append(img_boxes)
fig = plt.figure(figsize=(12,24))
visualize(fig, 8, 2, out_images, out_titles)
In [27]:
# Define a single function that can extract features using hog sub-sampling and make predictions
def find_cars(img, ystart=SW_YSTART, ystop=SW_YSTOP, scale=SW_SCALE, svc=SVC, X_scaler=X_SCALER,
orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS):
spatial_feat=SW_SPATIAL_FEAT_FLAG
hog_feat=SW_HOG_FEAT_FLAG
hist_feat=SW_COLOR_HIST_FEAT_FLAG
# If y start/stop positions not defined, set to image size
if ystart == None or ystart < 0:
ystart = 0
if ystop == None or ystop > img.shape[0]:
ystop = img.shape[0]
draw_img = np.copy(img)
#Make a heatmap of zeros
heatmap = np.zeros_like(img[:,:,0])
img = normalize_pixels(img).astype(np.float32)
img_to_search = img[ystart:ystop,:,:]
ctrans_tosearch = convert_color(img_to_search, conv=SW_CONVERT_COLOR)
if scale != 1:
imshape = ctrans_tosearch.shape
ctrans_tosearch = cv2.resize(ctrans_tosearch, (np.int(imshape[1]/scale), np.int(imshape[0]/scale)))
ch1 = ctrans_tosearch[:,:,0]
ch2 = ctrans_tosearch[:,:,1]
ch3 = ctrans_tosearch[:,:,2]
# Define blocks and steps as above
nxblocks = (ch1.shape[1] // pix_per_cell)-1
nyblocks = (ch1.shape[0] // pix_per_cell)-1
nfeat_per_block = orient*cell_per_block**2
window = HOG_PIXELS_PER_CELL*HOG_PIXELS_PER_CELL # 8 cells and 8 pix per cell
nblocks_per_window = (window // pix_per_cell)-1 # The // division is used for integers (for indices)
cells_per_step = 1 # Instead of overlap, define how many cells to step
nxsteps = (nxblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
nysteps = (nyblocks - nblocks_per_window) // cells_per_step # The // division is used for integers (for indices)
# Compute individual channel HOG features for the entire image
hog1 = get_hog_features(ch1, orient, pix_per_cell, cell_per_block, feature_vec=False)
hog2 = get_hog_features(ch2, orient, pix_per_cell, cell_per_block, feature_vec=False)
hog3 = get_hog_features(ch3, orient, pix_per_cell, cell_per_block, feature_vec=False)
for xb in range(nxsteps):
for yb in range(nysteps):
ypos = yb*cells_per_step
xpos = xb*cells_per_step
# Extract HOG for this particular patch
hog_feat1 = hog1[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_feat2 = hog2[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_feat3 = hog3[ypos:ypos+nblocks_per_window, xpos:xpos+nblocks_per_window].ravel()
hog_features = np.hstack((hog_feat1, hog_feat2, hog_feat3))
xleft = xpos*pix_per_cell
ytop = ypos*pix_per_cell
# Extract the image patch
subimg = cv2.resize(ctrans_tosearch[ytop:ytop+window, xleft:xleft+window], (DEFAULT_LENGTH, DEFAULT_WIDTH))
# Get color features
spatial_features = bin_spatial(subimg, size=BIN_SPATIAL_SIZE)
hist_features = color_hist(subimg, nbins=HIST_NBINS)
test_feats = np.hstack((spatial_features, hist_features, hog_features))
# Scale features and make a prediction
test_features = X_scaler.transform(test_feats.reshape(1, -1))
test_prediction = svc.predict(test_features)
if test_prediction == 1:
xbox_left = np.int(xleft*scale)
ytop_draw = np.int(ytop*scale)
win_draw = np.int(window*scale)
cv2.rectangle(draw_img,(xbox_left, ytop_draw+ystart),
(xbox_left+win_draw, ytop_draw+win_draw+ystart),BBOX_COLOR,6)
heatmap[ytop_draw+ystart:ytop_draw+win_draw+ystart, xbox_left:xbox_left+win_draw] +=1
return draw_img, heatmap
In [28]:
# DEFINE A CLASS TO RECEIVE THE CHARACTERISTICS OF EACH VEHICLE DETECTION
# Objects defined as "Vehicles" will be where multiple overlaping detections exists in the heatmap
class Vehicle():
def __init__(self, bbox):
self.prev_detected = True # Flag sets if the Vehicle was detected in the last iteration
self.n_detections = 1 # number of times this vehicle has been detected
self.n_non_detections = 0 # number of consecutive times this vehicle has not been detected
self.xpixels = np.arange(bbox[0][0], bbox[1][0]+1) # Pixel x values of last detection
self.ypixels = np.arange(bbox[0][1], bbox[1][1]+1) # Pixel y values of last detection
self.recent_xfitted = []
self.recent_xfitted.append(bbox[0][0]) # x position of the last n fits of the bounding box
self.bestx = bbox[0][0] # X position of the current fit
self.recent_yfitted = []
self.recent_yfitted.append(bbox[0][1]) # Y position of the last n fits of the bounding box
self.besty = bbox[0][1] # Average y position of the current fit
self.recent_wfitted = []
self.recent_wfitted.append(bbox[1][0])
self.bestw = bbox[1][0] # Average width of the last n fits
self.recent_hfitted = []
self.recent_hfitted.append(bbox[1][1]) # Height of the last n fits of the bounding box
self.besth = bbox[1][1] # Average height of the last n fits
self.bounding_box = bbox
In [29]:
from scipy.ndimage.measurements import label
def apply_threshold(heatmap, threshold):
# Zero out pixels below the threshold
heatmap[heatmap <= threshold] = 0
return heatmap
def draw_labeled_bboxes(img, labels, smoothing_factor=0, remove_threshold = 10):
# Iterate through all detected cars
for car_number in range(1, labels[1] + 1):
#Find pixels with each car_number label value
nonzero = (labels[0] == car_number).nonzero()
# Identify x and y values of those pixels
nonzeroy = np.array(nonzero[0])
nonzerox = np.array(nonzero[1])
# Define a bounding box based on min/max x and y
bbox = ((np.min(nonzerox), np.min(nonzeroy)), (np.max(nonzerox), np.max(nonzeroy)))
# Check if bounding box appears in carslist
found_match = False
for car in carslist:
# Check if car's previous pix values are 'close enough' to current bounding_box. If so, grab vehicle
if(bbox[0][0] in car.recent_xfitted and bbox[0][1] in car.recent_yfitted and
bbox[1][0] in car.recent_wfitted and bbox[1][1] in car.recent_hfitted):
found_match = True
print('Found a match. Car Number', car_number, '| length nonzerox:',len(nonzerox), '| length nonzeroy:',len(nonzeroy))
print('Bounding box:',bbox)
car.n_detections += 1
car.prev_detected = True
car.n_non_detections = 0 # Let's try setting this to 0 initially
car.xpixels = nonzerox # Pixel x values of current detection
car.ypixels = nonzeroy # Pixel y values of current detection
if len(car.recent_xfitted) >= smoothing_factor:
car.recent_xfitted.append(bbox[0][0])
car.bestx = int(np.mean(car.recent_xfitted[:-smoothing_factor])) # Average x position of the last n fits
car.recent_yfitted.append(bbox[0][1]) # Y position of the last n fits of the bounding box
car.besty = int(np.mean(car.recent_yfitted[:-smoothing_factor])) # Average y position of the current fit
car.recent_wfitted.append(bbox[1][0])
car.bestw = int(np.mean(car.recent_wfitted[:-smoothing_factor])) # Average width of the last n fits
car.recent_hfitted.append(bbox[1][1]) # Height of the last n fits of the bounding box
car.besth = int(np.mean(car.recent_hfitted[:-smoothing_factor])) # Average height of the last n fits
else:
car.recent_xfitted.append(bbox[0][0])
car.bestx = int(np.mean(car.recent_xfitted)) # Average x position of the last n fits
car.recent_yfitted.append(bbox[0][1]) # Y position of the last n fits of the bounding box
car.besty = int(np.mean(car.recent_yfitted)) # Average y position of the current fit
car.recent_wfitted.append(bbox[1][0])
car.bestw = int(np.mean(car.recent_wfitted)) # Average width of the last n fits
car.recent_hfitted.append(bbox[1][1]) # Height of the last n fits of the bounding box
car.besth = int(np.mean(car.recent_hfitted)) # Average height of the last n fits
car.bounding_box = ((car.bestx, car.besty),
(car.bestw, car.besth))
break
if found_match == False:
# Add New Vehicle
car = Vehicle(bbox)
car.xpixels = nonzerox
car.ypixels = nonzeroy
#Draw the box on the image
if (car.n_detections > smoothing_factor and car.n_non_detections == 0):
cv2.rectangle(img, (car.bestx, car.besty),(car.bestw, car.besth), BBOX_COLOR, BBOX_THICK)
# Add car to carslist
carslist.append(car)
# Set n_non_detections for each car in carslist that wasn't prev_detected
# n_non_detections = Number of consecutive times this car has not been detected
undetected_cars = (car for car in carslist if car.prev_detected is False and car.n_non_detections > 1)
for car in undetected_cars:
car.n_non_detections +=1
if car.n_non_detections > remove_threshold:
print('Removing Car: ', car.bounding_box)
carslist.remove(car)
# Return the image
return img
In [30]:
out_images = []
out_maps = []
out_titles = []
out_boxes = []
carslist = []
## Sliding Window Parameters - HOG Sub-Sampling ##
SW_CONVERT_COLOR = 'RGB2YCrCb'
SW_YSTART = None
SW_YSTOP = None
SW_SCALE = 1.75
BIN_SPATIAL_SIZE = (32,32)
HIST_NBINS = 128
HOG_ORIENTATIONS = 9
HOG_PIXELS_PER_CELL = 8
HOG_CELLS_PER_BLOCK = 2
HOG_CHANNEL = 'ALL' # Can be 0, 1, 2, or "ALL"
SW_SPATIAL_FEAT_FLAG = True
SW_HOG_FEAT_FLAG = True
SW_COLOR_HIST_FEAT_FLAG = True
#Iterate over the test images
for img_path in image_paths:
img = mpimg.imread(img_path)
out_img, heatmap = find_cars(img, ystart=SW_YSTART, ystop=SW_YSTOP, scale=SW_SCALE, svc=SVC, X_scaler=X_SCALER,
orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS)
labels = label(apply_threshold(heatmap, 2))
# Draw bounding boxes on a copy of the image
draw_img = draw_labeled_bboxes(np.copy(out_img), labels)
out_images.append(draw_img)
out_images.append(heatmap)
# filename = os.path.split(img_path)[-1]
out_titles.append(filename)
out_titles.append(filename)
fig = plt.figure(figsize=(12,24))
visualize(fig, 8, 2, out_images, out_titles)
In [61]:
def process_image(img):
## Search for cars from previous frame Gather previous bboxes from carslist
offset = 32
heatmap_1 = np.zeros_like(img[:,:,0])
out_img = img
if len(carslist) > 1:
bboxes = []
for car in carslist:
bboxes.append(((car.bestx, car.besty),(car.bestx+car.bestw, car.besty+car.besth)))
bboxes = np.array(bboxes)
# print('Found bboxes of size:', len(bboxes))
# print('bboxes[:,0]:', bboxes[:,0])
# Try sliding windows to capture wider bbox
# windows_1 = slide_windows(img, x_start_stops=bboxes[:,0], y_start_stops=bboxes[:,1],
# xy_windows=[(32,32)],
# xy_overlaps=[(0.85, 0.85)])
# _, heatmap_1 = search_windows(img, windows_1,svc=SVC, X_scaler=X_SCALER, color_space=COLOR_SPACE,
# spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS,
# orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
# hog_channel=HOG_CHANNEL, spatial_feat=SW_SPATIAL_FEAT_FLAG,
# hist_feat=SW_COLOR_HIST_FEAT_FLAG, hog_feat=SW_HOG_FEAT_FLAG)
# labels = label(apply_threshold(heatmap_1, 3))
# # # Draw bounding boxes on a copy of the image
# out_img = draw_labeled_bboxes(np.copy(img), labels, smoothing_factor=2)
## Search for cars using HOG subsampling (in current frame)
_, heatmap_2 = find_cars(img, ystart=SW_YSTART, ystop=SW_YSTOP, scale=SW_SCALE, svc=SVC, X_scaler=X_SCALER,
orient=HOG_ORIENTATIONS, pix_per_cell=HOG_PIXELS_PER_CELL, cell_per_block=HOG_CELLS_PER_BLOCK,
spatial_size=BIN_SPATIAL_SIZE, hist_bins=HIST_NBINS)
heatmap = np.add(heatmap_1, heatmap_2)
#print(heatmap)
labels = label(apply_threshold(heatmap_2, 3))
draw_img = draw_labeled_bboxes(np.copy(out_img), labels, smoothing_factor=2)
return draw_img
In [62]:
# carslist = []
# out_img = process_image(mpimg.imread(image_paths[0])) # For Debugging #
# plt.imshow(out_img)
In [ ]:
#Import packages to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
SW_YSTART = None #384
SW_YSTOP = None #640
SW_SCALE = 1.75
test_ouput = 'test_output.mp4'
VIDEO_OUTPUT_PATH = os.path.join(WORKING_DIRECTORY, test_ouput)
TEST_VIDEO = 'test_video.mp4'
PROJECT_VIDEO = 'project_video.mp4'
VIDEO_FILE_PATH = os.path.join(WORKING_DIRECTORY, TEST_VIDEO)
clip = VideoFileClip(VIDEO_FILE_PATH)
carslist = []
test_clip = clip.fl_image(process_image)
#%time
test_clip.write_videofile(VIDEO_OUTPUT_PATH, audio=False)
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(VIDEO_OUTPUT_PATH))
Out[ ]:
In [ ]:
project_ouput = 'project_output.mp4'
PROJECT_VIDEO = 'project_video.mp4'
VIDEO_FILE_PATH = os.path.join(WORKING_DIRECTORY, PROJECT_VIDEO)
VIDEO_OUTPUT_PATH = os.path.join(WORKING_DIRECTORY, project_ouput)
carslist = []
clip = VideoFileClip(VIDEO_FILE_PATH)
project_clip = clip.fl_image(process_image)
project_clip.write_videofile(VIDEO_OUTPUT_PATH, audio=False)
In [ ]:
HTML("""
<video width="960" height="540" controls>
<source src="{0}">
</video>
""".format(VIDEO_OUTPUT_PATH))
In [36]:
# # Add vehicle smoothing between frames
# carslist = []
# carslist.append(Vehicle())
# # Take the length of the least to know how many have been detected